{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f923a7f9-5505-4fbe-be4a-8870d9b2d7cf",
   "metadata": {},
   "source": [
    "## MultiOn Demo\n",
    "\n",
    "This notebook walks through an example of using LlamaIndex with MultiOn to browse the web on the users behalf.\n",
    "\n",
    "First, we import the OpenAIAgent that will control the Multion session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "daea8a25-468e-470c-99c9-ee6429bc522f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up OpenAI\n",
    "import openai\n",
    "from llama_index.agent import OpenAIAgent\n",
    "\n",
    "openai.api_key = \"sk-your-key\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e2a467-bb2b-4642-993d-80b1fc852add",
   "metadata": {},
   "source": [
    "We then import the MultiOn tool and initialize our agent with the tool. When the MultiOn tool is initialized, the user will be prompted to authenticate the session in their browser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "77d5b1ef-876f-4b90-94e4-dfde91e77fed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Already logged in.\n"
     ]
    }
   ],
   "source": [
    "# Set up MultiOn tool\n",
    "from llama_hub.tools.multion.base import MultionToolSpec\n",
    "\n",
    "multion_tool = MultionToolSpec()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d1d2f87-5a1f-4722-9f4f-ad0e072b190d",
   "metadata": {},
   "source": [
    "To support the MultiOn browsing session, we will also give our LlamaIndex agent a tool to search and summarize a users gmail inbox. We set up that tool below. For more information on the gmail tool, see the [Gmail notebook here](https://github.com/emptycrown/llama-hub/blob/main/llama_hub/tools/notebooks/gmail.ipynb).\n",
    "\n",
    "We will use this tool later on to allow the agent to gain more context around our emails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f74cab58-1b97-416e-bb7a-ee7eaa14555a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import and initialize our tool spec\n",
    "from llama_hub.tools.gmail.base import GmailToolSpec\n",
    "from llama_index.tools.ondemand_loader_tool import OnDemandLoaderTool\n",
    "\n",
    "# Initialize the Gmail tool to search our inbox\n",
    "gmail_tool = GmailToolSpec()\n",
    "\n",
    "# Wrap the tool so we don't overflow the main Agent's context window\n",
    "gmail_loader_tool = OnDemandLoaderTool.from_tool(\n",
    "    gmail_tool.to_tool_list()[1],\n",
    "    name=\"gmail_search\",\n",
    "    description=\"\"\"\n",
    "        This tool allows you to search the users gmail inbox and give directions for how to summarize or process the emails\n",
    "\n",
    "        You must always provide a query to filter the emails, as well as a query_str to process the retrieved emails.\n",
    "        All parameters are required\n",
    "        \n",
    "        If you need to reply to an email, ask this tool to build the reply directly\n",
    "        Examples:\n",
    "            query='from:adam subject:dinner', max_results=5, query_str='Where are adams favourite places to eat'\n",
    "            query='dentist appointment', max_results=1, query_str='When is the next dentist appointment'\n",
    "            query='to:jerry', max_results=1, query_str='summarize and then create a response email to jerrys latest email'\n",
    "            query='is:inbox', max_results=5, query_str='Summarize these emails'\n",
    "        \"\"\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee9b63f5-4f99-4708-b1ab-67d64fdb0cee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize our Agent with the MultiOn and Gmail loader tool\n",
    "agent = OpenAIAgent.from_tools(\n",
    "    [*multion_tool.to_tool_list(), gmail_loader_tool],\n",
    "    system_prompt=\"\"\"\n",
    "    You are an AI agent that assists the user in crafting email responses based on previous conversations.\n",
    "    \n",
    "    The gmail_search tool connects directly to an API to search and retrieve emails, and answer questions based on the content.\n",
    "    The browse tool allows you to control a web browser with natural language to complete arbitrary actions on the web.\n",
    "    \n",
    "    Use these two tools together to gain context on past emails and respond to conversations for the user.\n",
    "    \"\"\",\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d16f81c-ce52-4405-9bb7-e5950123df8b",
   "metadata": {},
   "source": [
    "Our agent is now set up and ready to browse the web!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a159dfe7-a3dc-4774-870c-23687611ffb1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Calling Function ===\n",
      "Calling function: browse with args: {\n",
      "  \"instruction\": \"browse to the latest email from Adam and open the email\"\n",
      "}\n",
      "running new session\n",
      "Server Disconnected. Please press connect in the Multion extension popup\n",
      "Got output: {'url': 'https://mail.google.com/mail/u/0/', 'status': 'CONTINUE', 'action_completed': 'I am navigating to Gmail to find the latest email from Adam.\\n\\n', 'content': 'Mail\\n\\nChat\\n\\nfo}\\nrR\\n\\nSpaces\\n\\nOr\\n\\nMeet\\n\\n4 Gmail\\n\\n2 Compose\\nfa Inbox\\n\\nyx © Starred\\n© _ Snoozed\\nE& Sent\\n\\nvy More\\nLabels\\n\\nQ Search in mail iz\\nOr ¢G i\\n\\nAdam, me 5\\n\\nMedium\\n\\nMedium\\n\\nEnable desktop notifications for Hypotenuse Mail.\\n\\nMedium Daily Digest\\nLinda Chen\\n\\nCanva\\n\\nZapier\\n\\nSlava Timbaliuc\\nSlava Timbaliuc\\nSlava Timbaliuc\\nMedium Daily Digest\\nSlava Timbaliuc\\nMedium - Programmi.\\n1Password\\n\\nYelp San Francisco\\nMedium Daily Digest\\nLucid\\n\\nDeb at Zapier\\n1Password\\n\\nMedium Daily Digest\\nsimon\\n\\nMedium Daily Digest\\n\\n@rctver @ @ 3 Google @\\n1-50 of 1,926 >\\nHerman Miller chair request - Hi, Okay ... 12:43PM\\nThe Edition: When you talk about Al, co... 10:14AM\\nStats for your stories: Sep 1-Sep 8 - A... 8:40AM\\nExperimenting Llamalndex RouterQuer... 7:40AM\\n\\nInvitation: Donut Friends! @ Wed Sep 1... [J Sep7\\n\\nPromote earth-friendly habits with this... Sep7\\n\\nMULTI-ON\\n\\nAuto | Step\\n\\nZapConnect\\n\\nUpdated invi\\nInvitation: Off\\n\\nUpdated invi Running on mail.google.com\\n\\nbrowse to the latest\\n\\nemail from Adam and\\nLeverage Al open the email\\nJoin yiding@\\nHow | made lam navigating to Gmail to\\nfind the latest email from\\n\\nAdam.\\n\\nUpdated invi\\n\\nSynapses—\\n'}\n",
      "========================\n",
      "=== Calling Function ===\n",
      "Calling function: browse with args: {\n",
      "  \"instruction\": \"open the latest email from Adam\"\n",
      "}\n",
      "session updated\n",
      "Got output: {'url': 'https://mail.google.com/mail/u/0/', 'status': 'CONTINUE', 'action_completed': 'I am clicking on the latest email from Adam to open it.\\n\\n', 'content': \"M4 Gmail Q Search in mail iz @rctver @ } 3: Google 6\\nG@ Compose € Oo w 4 © G&G D :  10f192% < >\\nInbox Herman Miller chair request (#ethal) inbox x > eG\\nyx © Starred\\n© _ Snoozed Adam Hofmann Hey, I'm interested in having one of these deliv... 12:29PM (1hour ago) +X\\n— Sent\\nv More\\n2\\nLabels +\\n\\nCa] Adam Hofmann. Hi, the price is firm at $480 +\\n\\nAdam Hofmann\\ntome v\\n\\nHi,\\n\\nOkay | can do $500 total for the chair and delive\\n\\n1513 St. Paul Street, St Catharines, Ontario, L2F\\n\\nYou can call me at 905-328-6945\\n\\nThanks\\n\\nSee you on Monday. | | Monday it is.\\n\\nEnable desktop notifications for Hypotenuse Mail.\\n\\nMULTI-ON\\n\\nAuto | Step.\\n\\nRunning on mail.google.com\\n\\nopen the latest email\\nfrom Adam\\n\\n| am clicking on the latest\\nemail from Adam to open it.\\n\"}\n",
      "========================\n",
      "I have opened the latest email from Adam for you.\n"
     ]
    }
   ],
   "source": [
    "print(agent.chat(\"browse to the latest email from adam and open the email\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cdf45a41-6009-46e5-a809-8d818c3b4d0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Calling Function ===\n",
      "Calling function: gmail_search with args: {\n",
      "  \"query\": \"from:adam\",\n",
      "  \"max_results\": 5,\n",
      "  \"query_str\": \"Summarize the email chain with Adam\"\n",
      "}\n",
      "Got output: The email chain with Adam consists of two emails. In the first email, the sender asks about the soonest delivery date for a chair and proposes a price of $475 with cash payment. In the second email, Adam agrees to the price of $500 for the chair and delivery, provides his address and phone number, and thanks the sender.\n",
      "========================\n",
      "Based on the summary of the email chain with Adam, here is a response that confirms all the details:\n",
      "\n",
      "\"Hi Adam,\n",
      "\n",
      "Thank you for your email. I wanted to confirm that I have received your message regarding the chair. I agree to the price of $500 for the chair and delivery. \n",
      "\n",
      "Here are the details:\n",
      "- Price: $500\n",
      "- Delivery address: 1513 St. Paul Street, St Catharines, Ontario, L2F\n",
      "- Phone number: 905-328-6945\n",
      "\n",
      "I appreciate your prompt response and I look forward to receiving the chair on Monday as discussed.\n",
      "\n",
      "Best regards,\n",
      "[Your Name]\"\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    agent.chat(\n",
    "        \"Summarize the email chain with adam and create a response to the last email\"\n",
    "        \" that confirms all the details\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "07fe11f9-da88-4eed-baa8-020a1e795cbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Calling Function ===\n",
      "Calling function: browse with args: {\n",
      "  \"instruction\": \"compose a reply email and paste the following content:\\n\\n\\\"Hi Adam,\\n\\nThank you for your email. I wanted to confirm that I have received your message regarding the chair. I agree to the price of $500 for the chair and delivery.\\n\\nHere are the details:\\n- Price: $500\\n- Delivery address: 1513 St. Paul Street, St Catharines, Ontario, L2F\\n- Phone number: 905-328-6945\\n\\nI appreciate your prompt response and I look forward to receiving the chair on Monday as discussed.\\n\\nBest regards,\\n[Your Name]\\\"\"\n",
      "}\n",
      "session updated\n",
      "Got output: {'url': 'https://mail.google.com/mail/u/0/', 'status': 'CONTINUE', 'action_completed': 'I am clicking on the \"Reply\" button to start composing the reply email to Adam Hofmann.\\n\\n', 'content': 'Mail\\n\\nChat\\n\\nfo}\\nrR\\n\\nSpaces\\n\\nOr\\n\\nMeet\\n\\nM4 Gmail Q. Search in mail iz @rctver @ } 3: Google 6\\n\\nG Compose € Oo w Y © G&G mo D : 10f1928 << >\\n2\\nfb Inbox\\nyr Starred Ca] Adam Hofmann Hi, the price is firm at $480 + $20 for delivery. ... 12:41PM (1hour ago) +\\n© _ Snoozed\\n—& Sent\\nvy Mor\\nore Adam Hofmann 12:43PM (1hourago) ye © :\\ntome v\\nLabels + regarding the char. |\\n\\nHi,\\n\\nagree to the price of\\n\\n$500 for the chair and\\n\\nOkay | can do $500 total for the chair and delive ,\\ndelivery.\\n\\n1513 St. Paul Street, St Catharines, Ontario, L2F :\\nHere are the details:\\n\\n- Price: $500\\n- Delivery address:\\n1513 St. Paul Street, St\\nCatharines, Ontario,\\n“ee L2F\\n- Phone number: 905-\\n328-6945\\n\\nCa] © vy Adam Hofmann (gmail.com) | appreciate your\\n\\nYou can call me at 905-328-6945\\n\\nThanks\\n\\nprompt response and |\\nlook forward to\\nreceiving the chair on\\nMonday as discussed.\\n\\noO Best regards,\\n[Your Name]\"\\n\\nEnable desktop notifications for Hypotenuse Mail.\\n\\nlam clicking on the \"Reply\"\\n\\nhittan $n ntart namnnninan\\n'}\n",
      "========================\n",
      "I have composed the reply email with the provided content. You can review and send the email by clicking on the \"Send\" button.\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    agent.chat(\n",
    "        \"pass the entire generated email to the browser and have it send the email as a\"\n",
    "        \" reply to the chain\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "494188f3-d314-4782-b4a8-06801440efac",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
